Skip to content

Match RSKj: pass its consensus tests, match every block of mainnet and testnet, handle forks and orphan blocks as RSKj does, and run faster - #374

Open
ColbySerpa wants to merge 268 commits into
Rustock-dev:mainfrom
HORNET-Storage:parity
Open

ColbySerpa wants to merge 268 commits into
Rustock-dev:mainfrom
HORNET-Storage:parity

Conversation

@ColbySerpa

Copy link
Copy Markdown

RSKj's own tests of the network's rules pass on rustock, every block of mainnet and testnet matches RSKj, and rustock handles forks and orphan blocks as RSKj does. It is also faster than RSKj on every measure we took. All of it is held to RSKj's release VETIVER-9.0.4 and the bitcoinj-thin it builds with (0.14.4-rsk-18).

RSKj's tests pass

RSKj's tests that name one of the network's rules or upgrades, 1,239 of them, were each read, and each has its verdict in docs/rskj-tests/:

RSKj's tests of the network's rules
Ported to rustock, and passing 925
Ported, found rustock doing otherwise, rustock fixed, now passing 13
Held by a test rustock already had, which passes 206
Testing what has no counterpart in rustock: Java's nulls and constructor checks, RSKj's mocks and wiring, calls RSKj never makes 94
A class only in RSKj's master, not in its release 1

So every one that applies to rustock passes: 1,144 of 1,144. The 248 tests of RSKj's validators, miner and transaction pool are held the same way, with none left a gap (docs/rskj-test-parity.md).

Every block matches RSKj

As CLAUDE.md asks of a consensus change, this was checked against the chain, not reasoned about. Every block of both networks was replayed: mainnet from genesis to #9,420,342 and testnet from genesis to #8,306,564. Every block matched what its header records, its state root included wherever the header holds the unitrie's (all of testnet, and mainnet from #1,591,000), and each block the run ended on matches RSKj's public node, its hash and its state root. No block was refused, no node halted, and no node logged an error. The build was this PR's but for its last ten commits and upstream's newest main, merged in since. The ten change only what a mining node builds and answers (the gas limit and the minimum gas price it aims for, its answers to a pool, its timed miner and its development clock, evm_increaseTime) and a check at start-up, or add only documents and tests; upstream's brings the external-IP lookup, Docker files, a newer libz-sys and a section of CLAUDE.md. None of it changes how a node checks or runs a block it receives.

Five testnet blocks that rustock got wrong now replay to their headers' state and receipts roots. Each was a network rule, and each is fixed with a test that fails without it:

Block What RSKj does there
#2,122,593 Before RSKIP-284, reads a Flyover BTC address's version byte as a signed Java byte, so bitcoinj refuses 0xc4 and the Bridge answers its generic error
#2,237,507 Can't size a peg-out input spent with testnet's early hard-coded ERP redeem script (bitcoinj throws), so updateCollections ends with nothing stored
#3,088,983 Keeps a nested constructor's self-destruct refund, 24,000 gas
#4,513,687 Turns down a peg-in it can't refund (rejected_pegin, then unrefundable_pegin); two more of RSKj's peg-in rules from RSKIP-379 on are fixed with it
#5,214,732 Leaves the creator's return data empty after a reverted creation

Faster than RSKj

rustock with this PR (at 8a00981), against RSKj VETIVER-9.0.4, on the same machine and the same mainnet blocks, all measured on 17 November:

rustock RSKj rustock vs RSKj
Catching up from the network: 6,907 blocks, start to finish¹ 57 s 18 min 19× faster
Loading Rootstock's signed copy with --import² 18.5 s 59.9 s 3.2× faster
Memory while re-running the blocks³ 0.45 GB 1.26 GB 2.8× less
CPU time to re-run the blocks³ 21.8 s 42.6 s 2× less
Time per block inside the node while catching up⁴ 3.8 ms 6.1 ms 1.6× faster
Re-running the blocks offline³ 20.8 s 26.9 s 1.3× faster

In every run, every block matched its header's state root on both clients.

¹ From the moment each node had Rootstock's copy of mainnet #9,414,000 loaded (the end of each one's --import) to running block #9,420,907, the newest block when rustock had caught up, fetching the blocks from mainnet peers. RSKj took 18 min 0 s, rustock 57 s. Most of the gap is in fetching: rustock had all 6,907 blocks 46 s after its load, from up to 14 peers with up to 176 bodies in flight, while RSKj took in about 4.4 blocks a second for its first 14 minutes.

² The copy at mainnet #9,414,000, which all three signers list with the same hash (both clients trust the same three): 4,000 blocks, 781,768 long values and 10,683,761 state nodes. Both clients read the signers' lists, check their signatures and the file's SHA-256, and load it; the times are the load alone, after the download. RSKj's 59.9 s is by its own log, from after it unzipped the file; rustock's 18.5 s include unzipping it.

³ #9,405,001 to #9,412,300 (7,300 blocks), re-run from each client's own database with every state root checked against its header, and a cache of a million trie nodes in front of each (RSKj's default cache.states.max-elements): rustock with replay_bench (crates/cli/examples), RSKj with its own ExecuteBlocks tool. Four runs each, alternating, each client twice in a row, after a first pass that brought both databases into memory. Time is the average re-run: rustock's leaves out opening its store, and RSKj's leaves out the 0.9 s its tool takes for a single block. CPU is the whole process's, user and system. Memory is the highest peak resident set of each client's four runs.

⁴ The same stretch as ¹, from each node's log: for rustock, the time it spent executing and storing the blocks (26.3 s for 6,907); for RSKj, the time inside tryToConnect, which also validates them (42.2 s for 6,907).

Forks and orphan blocks, as RSKj handles them

Every block is executed as it arrives, against its own parent's state, beside the chain when it isn't on it. A heavier branch becomes the best only once this node has executed it, and switching to it re-points the chain and runs nothing again. An orphan block, one whose parent this node lacks, has the missing blocks asked of the peer that sent it, as RSKj asks, and its branch is executed as they come. A branch none of whose bodies arrive within 60 s is given up, so withholding bodies can't hold the node back. Before, a heavier branch known only by its headers took the head and pulled execution back to the fork point, where the miner built and every query was answered until its bodies came and it was re-run.

On a tie in total difficulty, the block store and the miner apply RSKj's SelectionRule (fees, then the lower hash), so a miner on rustock builds on the same block as RSKj's miners.

At the mainnet tip for four hours, as competing blocks met, the node ran 808 blocks beside the chain and switched to a heavier branch 333 times, running nothing again, and at each of 25 checks it was level with RSKj's public node.

See docs/follow-mode-rskj-vs-rustock.md §8, and §99 and §100 of docs/design-divergences.md, which records RSK's rules where they differ from Ethereum's.

Parity with RSKj, RSKIP by RSKIP

The 108 RSKIPs below were each read side by side in RSKj's release VETIVER-9.0.4 and in rustock, each under the upgrade that brings it: the code of each in both clients, with RSKj's own tests of it as the checklist. So was each statement rsk.zone makes about what a node does: merge mining, the PowPeg, REMASC, the EVM and its precompiles, the state tree, the chain IDs. Where rustock did otherwise than RSKj it now does as RSKj does, and each fix comes with a test that fails without it: 74 match RSKj, some of them through this PR's other fixes; 30 had differences, all fixed; and 4 have nothing for a node to implement. Where a row says so, RSKj confirmed the fix on its own regtest: a case builds blocks on RSKj's release, the replay's build imports them with every state root checked against its header and agrees, and a build from before the fix parts from RSKj where the fix says it would.

Every other RSKIP rsk.zone lists was read against RSKj as well, and each statement it makes about what a node does holds of rustock as it holds of RSKj.

Before any upgrade (from block 0)

RSKIP What it makes a node do What was found
0 The RSKIPs' purpose and guidelines Nothing for a node to implement, in either client.
15 REMASC: a block's fees paid to its miner 4,000 blocks later, shared with its siblings Three fixes. REMASC pays the matured block of the running block's own chain, and its siblings, in RSKj's order; rustock took them from the main chain by number, which shows only on a block run beside the chain (confirmed on RSKj's regtest). Its payment to the federation takes a new federation's activation age at the running block's upgrades, not the paid block's (confirmed on RSKj's regtest). A block or body the store lacks, when REMASC reads it, is the node's fault and no rule's; rustock refused the block (no regtest reaches it: a store fault).
40 The Bridge: peg-ins, the federation and its changes, peg-outs and their signatures Fixed in each of its three parts. Bitcoin headers: before RSKIP-88, receiveHeaders fails for a header that counts its transactions below zero (no regtest reaches it: before the first upgrades, where RSKj's regtest cannot start); a Bitcoin block looked up by hash is read from the Bridge's storage alone, not from a node-wide cache that keeps other chains' blocks (confirmed on RSKj's regtest). The federation: with no federation pending, a vote to add a member whose key is no point answers -10 (confirmed on RSKj's regtest); a chain from RSKj's regtest genesis has its five authorizers, three deciding, and its ten free Bridge senders (confirmed on RSKj's regtest); a member key of the one byte 00 is kept, as RSKj keeps it (confirmed on RSKj's regtest); a migration halves the balance with no floor and, under dust, fails inside the migration age and retires the federation past it (confirmed on RSKj's regtest); and it pays the active federation as stored, in its own format (confirmed on RSKj's regtest). Peg-outs: a signature is read as bitcoinj reads it with Spongy Castle 1.58, every element of it; a high s signs nothing; the input gets the DER bitcoinj writes (confirmed on RSKj's regtest). The confirmed peg-out taken is the first in the walk order of the set the call read (confirmed on RSKj's regtest). addSignature writes back the set it read (confirmed on RSKj's regtest), logs the member's key compressed (confirmed on RSKj's regtest), and fails where an input has no redeem script (no regtest reaches it: the Bridge builds every transaction it signs). The federations' UTXOs are written back where RSKj reads them: by the fee estimate and by a peg-in turned down (confirmed on RSKj's regtest). Before RSKIP-88, a call answered with null saves nothing, and the state getters write back what they read (no regtest reaches it: before the first upgrades).
42 Receipts with no intermediate state root Matches RSKj.
46 Mining fees in the header Matches RSKj.
50 A code header in front of a contract's code Fixed: code of four bytes or more that starts with its header (0xfc) runs from byte 4 + code[3], its jump destinations found from there, as RSKj runs every program from genesis; rustock ran it from byte 0 and failed it at the 0xfc with all its gas. Anyone can deploy such code, so one transaction calling it would have split rustock from the chain. The test is built from RSKj's own VMTest cases.
59 Child contracts Nothing for a node to implement, in either client.
107 The unitrie's smaller nodes, as stored and hashed Fixed: a tree node with bytes after its long value's length is refused (no regtest reaches it: no client writes one; a peer could send one in a snapshot).
148 The ERC1820 registry Nothing for a node to implement, in either client.
157 Cumulative difficulty in JSON-RPC Matches RSKj (RPC only).
159 A minimal proxy contract Nothing for a node to implement, in either client.
225 The emergency keys of the federation's time-locked script Matches RSKj.

Bahamas (mainnet #3,397, testnet #0)

RSKIP What it makes a node do What was found
136 How a transaction's own call to a precompile is charged Fixed: before it, RSKj holds such a call to its limit by the precompile's price alone, and charges the price on top of the intrinsic cost, past the limit if it comes to that; rustock did so for the Bridge only. Three more fixes in the same path, which also takes the federation's free Bridge transactions: a transaction is held to what TransactionExecutor.init checks (the sender's nonce, a limit that pays the intrinsic cost, a balance that covers the value and the limit), and a block holding one that fails it is refused, as RSKj refuses it, so the upstream test test_bridge_tx_gas0_fails_for_unknown_sender now asserts the refusal; a byte of data costs 16 gas from RSKIP-400, not 68; and where a block runs as its sublists, REMASC is paid each fee once, not twice.

Orchid (mainnet #729,000, testnet #0)

RSKIP What it makes a node do What was found
85 REMASC holds back a reward under the minimum payable Fixed: REMASC holds back a reward under the block's whole minimum gas price times its minimum payable gas; rustock stopped the price at 2^64 − 1 (confirmed on RSKj's regtest, at a price of 2^65).
87 The lock whitelist's unlimited mode Matches RSKj.
88 The Bridge refusing calls from contracts, and how it ends a call it can't read Matches RSKj; how the Bridge ends a call before it is under RSKIP-40.
89 The Bridge's Bitcoin block query methods Matches RSKj.
90 A precompile's output as the caller's return data Fixed: before it, a precompile that returns leaves the caller's return data as it was; rustock made the output the return data (no block of mainnet or testnet reaches it).
91 STATICCALL Matches RSKj.
92 The merged-mining merkle proof's encoding Matches RSKj.
97 The difficulty's ten-minute rule and its minimum Matches RSKj.
98 No federated fallback mining Matches RSKj.

Orchid 0.6.0 (mainnet #1,052,700, testnet #0)

RSKIP What it makes a node do What was found
103 STATICCALL with no value Fixed: before it, STATICCALL takes a value, as CALL does, with CALL's charge and stipend for it; rustock took no value (no block of mainnet or testnet reaches it).

Wasabi (mainnet #1,591,000, testnet #0)

RSKIP What it makes a node do What was found
106 The HD wallet precompile Fixed: a derivation whose HMAC's left half equals the curve's order keeps its parent's point; rustock threw (no regtest reaches it: it needs such an HMAC output).
110 The fork-detection data in a merged-mining coinbase Matches RSKj.
119 The BlockHeader precompile Matches RSKj.
120 The shift opcodes Matches RSKj.
122 The Bridge's getBtcTransactionConfirmations, its price included Matches RSKj.
123 Federation members with several keys Fixed: the new federation is written back after any successful call that read it (no regtest reaches it: before Wasabi, where RSKj's regtest cannot start).
124 The Bridge's receiveHeaders: its price, who may call it, a header's size Matches RSKj.
125 CREATE2 Matches RSKj.
126 The state root a header carries, and how a block's roots are checked Matches RSKj.
131 CREATE2 after SUICIDE in the same block Fixed: a CREATE2 at an address deleted earlier in the same block fails, as RSKj refuses it; rustock let it succeed.
132 A higher gas cost for receiveHeaders Matches RSKj.

TwoToThree (mainnet #2,018,000, testnet #504,000)

RSKIP What it makes a node do What was found
150 A call's stipend added to the gas it asks for Fixed: before it, a call that asks for 2^63 − 2,300 gas or more and sends a value overflows a Java long, and the calling frame fails with all its gas; rustock saturated the sum (no block of mainnet or testnet reaches it).

Papyrus (mainnet #2,392,700, testnet #863,000)

RSKIP What it makes a node do What was found
134 Raising the locking cap Fixed: increaseLockingCap fails the call where the new cap times its multiplier passes a long; rustock saturated the product and went on (no authorizer has asked for such a cap).
137 The altBN128 precompiles, and their answer to a point off the curve before RSKIP-197 Matches RSKj.
140 EXTCODEHASH Matches RSKj.
143 The Bridge's coinbase-information methods Fixed: the witness root registered for a Bitcoin block is read only from it (no regtest reaches it: a coinbase is registered only from RSKIP-143).
146 The Bridge's events in Solidity's encoding Fixed: the release request queue is written back, both its cells, after every method that read it (confirmed on RSKj's regtest); getStateForDebugging leaves out the entries of the other encoding (confirmed on RSKj's regtest).
151 SELFBALANCE Matches RSKj.
152 CHAINID Matches RSKj.
156 The difficulty's bound divisor, 400 from it except on regtest Fixed: regtest keeps its divisor of 2,048 under it (confirmed on RSKj's regtest).

Iris (mainnet #3,614,800, testnet #2,060,500)

RSKIP What it makes a node do What was found
153 The BLAKE2 compression precompile Matches RSKj.
169 EXTCODEHASH rectified Fixed: before it, EXTCODEHASH of a contract whose empty code a creation of the same block saved answers keccak256 of nothing; rustock answered 0 (confirmed on RSKj's regtest).
170 Peg-ins to any address Fixed: a peg-in with an output bitcoinj can't parse is refunded; rustock credited it (confirmed on RSKj's regtest).
171 The return buffer cleared by the call opcodes Matches RSKj.
174 A creation keeps the balance at its address Fixed: before it, a deployment that fails leaves the balance at its address as it was (confirmed on RSKj's regtest); local calls, traces and estimates apply it as a block's run does (confirmed on RSKj's regtest).
176 Flyover: peg-ins to a programmable address Fixed: before RSKIP-293, a Flyover payment with an output that is no address answers GENERIC_ERROR; rustock answered -304, or registered it (confirmed on RSKj's regtest).
179 BTC-RSK timestamp linking Matches RSKj.
181 Peg-in rejection events Matches RSKj.
185 Peg-out refunds and events Fixed: before it, a contract's releaseBtc fails the contract that made the call, with all its gas; rustock failed only the call (confirmed on RSKj's regtest).
186 The active federation's creation block height Fixed: the getter answers the next federation's creation block once it is active, before updateCollections moves it (confirmed on RSKj's regtest).
191 DUPN, SWAPN and TXINDEX removed Fixed: before it, RSK's own opcodes DUPN, SWAPN and TXINDEX run as RSKj runs them; rustock had none of them.
197 Precompile calls with CALL's semantics Matches RSKj; its window with RSKIP-185 is under that one.
199 registerBtcTransaction open to anyone Matches RSKj.
200 Limits on receiveHeaders Matches RSKj.
201 The federation's time-locked emergency script Fixed: below the peg-out index, a registration is sorted by running each input's script as bitcoinj does, signatures checked; rustock compared the hash of the redeem script an input carried (confirmed on RSKj's regtest).
218 A new fee address for the Core Developers Fund Matches RSKj.
219 New minimum values for peg-ins and peg-outs Matches RSKj.
220 A Bitcoin block or header by hash or height in the Bridge's methods Matches RSKj.

Hop (mainnet #4,598,500, testnet #3,103,000)

RSKIP What it makes a node do What was found
271 Peg-outs in batches Fixed: a request must cover the fee of a peg-out sized by the active federation's format (confirmed on RSKj's regtest); a batch is halved while it is over 100,000 bytes (no regtest reaches it: it needs hundreds of peg-ins); a batched request with no hash fails the call (confirmed on RSKj's regtest); the next peg-out height is written back (confirmed on RSKj's regtest).
284 Flyover refund addresses read anew Matches RSKj; how an address is read before it is the fix for testnet block #2,122,593, above.
290 Testnet's minimum difficulty Matches RSKj; testnet's headers are held to it under Consensus, below.
293 getActivePowpegRedeemScript, and more Flyover checks Matches RSKj; the Flyover check before it is under RSKIP-176.
294 At most so many inputs in a migration Matches RSKj.
297 A wider Bitcoin-to-RSK timestamp gap on testnet Matches RSKj.

Hop 4.0.1 (mainnet #4,976,300, testnet #3,362,200)

RSKIP What it makes a node do What was found
353 Redeem scripts held to Bitcoin Core's standard checks Matches RSKj.
357 The confirmations of a migration period adjusted Matches RSKj.

Fingerroot (mainnet #5,468,000, testnet #4,015,800)

RSKIP What it makes a node do What was found
252 The transaction gas price cap Matches RSKj.
326 Peg-out events improved Matches RSKj.
374 The confirmations of a migration period restored Matches RSKj.
375 Peg-outs waiting for signatures keyed by their creation transaction's hash Fixed: a confirmed peg-out under a key the set waiting for signatures already holds, or under none, fails updateCollections; rustock replaced the entry, or used the update's own hash (confirmed on RSKj's regtest, both).
377 The last retired federation's standard P2SH script kept Matches RSKj.
383 A longer PowPeg activation age Matches RSKj.
385 getEstimatedFeesForNextPegOutEvent improved Matches RSKj.

Arrowhead (mainnet #6,223,700, testnet #4,927,100)

RSKIP What it makes a node do What was found
203 The getCallStackDepth precompile method Matches RSKj.
376 Migration transactions of version 2 Matches RSKj.
379 An index of peg-out and migration transactions Matches RSKj; how a registration is classified by its index is under Consensus, below, and two of the peg-in rules it brings are fixed with testnet block #4,513,687, above.
398 PUSH0 Matches RSKj.
400 Cheaper calldata Matches RSKj; its price of data in a transaction's own call to a precompile is under RSKIP-136.
412 BASEFEE Matches RSKj.
415 Pegnatories' addresses derived rightly from their keys Matches RSKj.
417 A Bridge call from a contract that answers an error no longer reverts it Matches RSKj.

Arrowhead 6.3.1 (mainnet #6,549,300, not on testnet)

RSKIP What it makes a node do What was found
434 Bitcoin chain work of up to 12 unsigned bytes Matches RSKj.

Lovell (mainnet #7,338,024, testnet #6,110,487)

RSKIP What it makes a node do What was found
419 The PowPeg's spendability validation protocol (SVP) Fixed: createFederation answers -4 while a proposed federation awaits its validation (confirmed on RSKj's regtest); a call to sign the SVP spend transaction with a key of no proposed member fails, and one with a member's key in its hybrid form signs (confirmed on RSKj's regtest).
427 Peg-out events' amounts in wei Matches RSKj.
428 A peg-out creation event with its UTXOs' values Matches RSKj.
438 Initcode's size limit and its gas per word Matches RSKj.
445 MCOPY Matches RSKj.
446 Transient storage Matches RSKj.
453 No account made by a failed CREATE or CREATE2 Matches RSKj.
454 Bitcoin chain work of up to 32 unsigned bytes Fixed: before it, a checkpoint whose chain work is past 12 bytes is refused, as bitcoinj refuses to write it; rustock kept its low 12 bytes (no regtest reaches it: RSKj's regtest seeds no checkpoint).
459 Rejected peg-ins marked processed Matches RSKj.
460 Non-standard outputs skipped in finding the witness commitment Matches RSKj.

Reed (mainnet #8,052,200, testnet #6,835,700)

RSKIP What it makes a node do What was found
305 Segwit peg-outs Fixed: a segwit input is signed only over the value kept for it, and only from RSKIP-305; before it, or with none kept, the call fails; rustock signed over a value of 0 (confirmed on RSKj's regtest).
516 Addition and multiplication on secp256k1 as precompiles Matches RSKj; its input cut is under Consensus, below.

Reed 8.1.0 (testnet #7,139,600, not on mainnet)

RSKIP What it makes a node do What was found
144 Parallel transaction execution Matches RSKj; REMASC's fees where a block runs as its sublists are under RSKIP-136.
351 A block's sublist gas limits and their header field Matches RSKj.
502 The PowPeg and Union Bridge integration Fixed: rustock made its seven Bridge methods callable on mainnet from #8,804,200, where RSKj has none of them, so one transaction calling one would have split rustock from the chain.
529 The Union Bridge's event methods Matches RSKj.
536 The BlockHeader precompile's getCumulativeWork and getDifficultyWithUncles Matches RSKj.

Vetiver (mainnet #8,804,200, testnet #7,604,200)

RSKIP What it makes a node do What was found
535 The header's version 2, and the Union Bridge's base event it carries (testnet only) Matches RSKj.
540 The Bridge's getEstimatedFeesForPegOutAmount Matches RSKj.
544 New code that starts with 0xEF refused Matches RSKj.
551 Where a rejected peg-in stops being marked as processed Matches RSKj.
552 BLAKE2F's price for a call with no data Matches RSKj.

tbd1000 (scheduled on no network)

RSKIP What it makes a node do What was found
559 Sorting the peg-outs Matches RSKj.

Other rules

Where What was found
The EVM A 32-byte word with bits set above an address's 20 bytes names no precompile, for a call, EXTCODESIZE or EXTCODEHASH (confirmed on RSKj's regtest). Stored code that starts 0xEF01 is code like any other, not a pointer (confirmed on RSKj's regtest). A transaction from an address that holds code runs (no regtest reaches it: it needs a key for such an address). BLOCKHASH answers from the running block's own ancestors, so a block beside the chain reads its own branch (confirmed on RSKj's regtest).
Block 0 and the genesis file Block 0 hashes over its difficulty as its file writes it, and a zero coinbase or gas limit as RSKj hashes them (confirmed on RSKj's regtest). A genesis file is read as RSKj reads it: a timestamp or gas limit over 8 bytes, or no minimum gas price, refused (confirmed on RSKj's regtest).
The Bridge A free Bridge transaction's receipt keeps what its Bridge call logged before it threw (confirmed on RSKj's regtest).
The miner A solved block a rule refuses is answered INVALID_BLOCK (confirmed on RSKj's regtest), and so is one the node fails to store (no regtest reaches it: a store fault). A solution of work whose parent is no longer the head is connected as any block (confirmed on RSKj's regtest). Uncles are picked by the rule a block meets on import, and a block it solves is held to every rule, its fork detection data included; a gas limit target under the minimum is the minimum; a timed miner started again right after a stop runs one thread; evm_increaseTime ignores zero or fewer seconds (no regtest reaches it: what a node mines is its own). And the miner has the settings of RSKj's it lacked, with RSKj's tests of them ported: the gas limit it aims for (targetgaslimit, 6,800,000 unless set and 7,500,000 on a regtest) and the switch that, off, steers by the gas the parent used (forcegaslimit); and a minimum gas price held stable in a currency (miner.stableGasPrice), fetched from a web API or from a contract.
Sync and relay A node on a new database runs block 1 in follow mode as it arrives (confirmed on RSKj's regtest). A block a peer sends whole, timed no later than its parent, is not relayed. Snap sync holds its blocks to the rules RSKj's snap client runs. An old store's block 0 is written again (no regtest reaches it: what a node relays and keeps).

What this PR does

It brings rustock in line with RSKj rule by rule, held to RSKj's release VETIVER-9.0.4 and the bitcoinj-thin it builds with (0.14.4-rsk-18), and then replays both networks from their first block to show it. It reads 108 RSKIPs side by side with RSKj, with RSKj's own tests ported as the checklist, and fixes each one that differed (above). It also gives rustock RSKj's --import and the miner settings of RSKj's it lacked. It is 264 commits, each one change, on upstream's main (640dfb9, merged in).

Closes #149.

Consensus

  • Native contracts run as RSKj's NativeContract: one argument decoder, gas charged before the method runs, and a throw settled as RSKj settles it.
  • Transactions, headers, gas limits and minimum gas prices are read, hashed, kept and written as RSKj does, by each network's activation heights, including headers that arrive compressed.
  • Headers are held to RSKj's rules: uncles, trunk timestamps and proof of work, fork detection, merged mining below Orchid, fallback-mined blocks' signatures, RSKIP-290's testnet minimum difficulty, and Java int arithmetic where RSKj uses it.
  • The Bridge decodes arguments as RSKj's decoder does, validates peg-ins in its order, holds Bitcoin headers to bitcoinj-thin's rules, classifies registered transactions by the peg-out index from RSKIP-379 and by the legacy rules below it, and seeds the BTC store as ensureBtcBlockStore does.
  • Precompiles: ECRecover, ModExp, BLAKE2F and RSKIP-516's input cut run as RSKj runs them, and what a precompile writes outside revm's journal goes back with a frame that fails.
  • RSKj's regtest: its schedule, Bridge constants, genesis federation and free senders.

Where the network's rule is simply what RSKj's code does, rustock does the same: docs/quirks-frozen-bugs.md records the RSKj behaviours that mainnet's and testnet's blocks depend on, and docs/design-divergences.md the places where RSK departs from Ethereum on purpose, each with the RSKj code rustock follows.

Nothing a stranger sends can panic the node

Everything a stranger can send was tried against it. A panic in a peer's session now ends that session and drops the peer, and nothing else. A peer offering a snapshot can no longer make the node hang, abort or grow its memory without limit. An RPC method that panics answers with an error, where the connection used to drop. Fuzz tests cover every RSK and P2P message, headers of every version, transactions, receipts, RLP, Bitcoin blocks and partial merkle trees, trie nodes and RSKj's snapshot chunks, and every precompile.

Sync and snapshot sync

  • Snapshot sync: the work a peer can make the node do is bounded (the header walk, the sampler, the rate limiter), a refused claim and headers nobody asked for stay out of the chain, RSKj's chunks are placed in the trie, and a node that has executed nothing tries the snapshot again instead of falling back to ordinary sync.
  • Ordinary sync: every block gets its own total difficulty with its uncles, a block sent whole is weighed with the uncles it brings, new blocks are announced, relayed and served as RSKj does, and RSKj's peer.bannedMiners is honoured.
  • The pool takes or refuses a sent transaction as RSKj's does, and reckons its cost in whole numbers.

--import, as RSKj's

rustock --import starts a node from the state copies Rootstock publishes for RSKj's --import, as RSKj's co.rsk.db.importer does: it reads each trusted signer's index.json, takes the highest height that more than half of the keys, and at least two, list with the same hash, checks their signatures over it, downloads the copy and checks its SHA-256, and loads its blocks and state with every check RSKj's reader makes, in both of the copy's formats. Mainnet and testnet use RSKj's URL and keys; another network names its own with --import-url and --import-trusted-key.

Where it differs from RSKj, it refuses rather than overwrites, and checks more. A data directory that already holds a chain is refused, where RSKj deletes it. If one signer's copy can't be fetched or doesn't hash right, the next one's is tried. The copy's blocks must form one chain, match their bodies, and end at the signed height with that block's state root among its nodes. Below the copy's lowest block, the chain is recorded as a pruned database records what it has dropped, so nothing looks for those blocks.

Run from an empty data directory against Rootstock's server: 3 of 3 signatures on #9,414,000, the 735 MB copy downloaded in 155 s and loaded in 18.5 s, and the node then caught up with mainnet, 6,907 blocks, in 57 s, to a block that matches RSKj's public node. A second --import into the same directory is refused, and a restart without it resumes.

RPC

Answers as RSKj gives them: a block's fields as BlockResultDTO prints them, block parameters and call arguments as RSKj reads them, eth_gasPrice computed in doubles as RSKj computes it, the methods that sign, traces (REMASC's and the Bridge's transfers as CALL subtraces, and traces and estimates run through the chain's own execution loop), the methods RSKj stubs or refuses, and a transaction answered from the main chain only.

For light clients, which check every byte against a block hash they trust: rsk_getStorageNodes, which proves a storage slot absent as well as present; rsk_getStateNodes, the trie walks along any keys; rsk_getRawBlocks, blocks as RSKj sends them to its peers; and rsk_getBlockNumberByHash.

macOS and Windows

The node runs on a thread with an 8 MiB stack on every OS, flatdb locks and reserves its files on Windows, a thread's CPU time is measured on macOS and Windows too, the alerts build on Windows, and the external-IP lookup's tests listen where localhost resolves first, since Windows takes two seconds to refuse a connection. CI builds and runs the whole test suite on macOS and Windows as well as Linux.

Changes a user may notice (not consensus changes)

None of these changes the network's rules: they are about how the node is started and configured, what its RPC answers, and how its miner works.

  • --import is new, with --import-url and --import-trusted-key for a network other than mainnet and testnet. They are command-line only: a config file can't set them, since a file that turned --import on would refuse every start after the first.
  • A new data directory starts on the default --trie-backend single. Upstream's check against a trie that isn't there (0c83af7) runs after block 0's header is written and before its state is built, so it refused every new database but one --import filled; it now refuses an empty trie only where blocks have run.
  • --max-gas-limit and [chain] max_gas_limit are gone, with the bound they lifted. RSKj has no maximum gas limit, and rustock refused gas limits above 1,000,000,000 (10,000,000 on regtest), which a gas limit raised as fast as the rules allow passes in about 5,100 blocks. A node given the flag now stops on it, so perf-lab's seed flags, which pass u64::MAX, need to drop it.
  • eth_call, eth_estimateGas and eth_sendTransaction take RSKj's defaults: a gas price of 0 when none is given, and 90,000 gas for a sent transaction. rustock took the block's minimum gas price for a call, and the tracker's price and the head's whole gas limit for a sent transaction. With RSKj's defaults, the same request gets the same answer from both.
  • eth_call is held to RSKj's call gas cap, 10,000,000, not the block's gas limit.
  • --hardfork refuses a schedule rustock would run otherwise than RSKj: one that leaves an upgrade out below an upgrade it schedules, or puts upgrades out of their order, or moves hop401, bahamas or orchid060, which rustock reads at one height per network. RSKj reads each upgrade's rules at that upgrade's own height, and rustock reads them from the upgrade in force, so before this it ran such a schedule silently otherwise than RSKj. No network has one.
  • A merged-mining pool gets RSKj's answer to a solution. A solved block that a rule or its run refuses, or that the node fails to store, is answered OK with INVALID_BLOCK, its hash and its height, and one whose parent the node lacks with NO_PARENT, where rustock failed the submission with an error; a merged-mining proof that fails is still an error, as in RSKj. A solution of work whose parent is no longer the head is run beside the chain, stored, and answered IMPORTED_NOT_BEST, or IMPORTED_BEST where it takes the head, where rustock refused it and the block was lost.
  • A mining node steers its blocks' gas limit as RSKj's does, by at most a 1024th of the parent's per block: toward --mining-gas-limit-target (gas_limit_target under [mining]), RSKj's targetgaslimit, 6,800,000 unless set and 7,500,000 on a regtest; and with --mining-force-gas-limit false (force_gas_limit), RSKj's forcegaslimit off, by the gas the parent used. rustock's miner kept the parent's gas limit.
  • A mining node's minimum gas price steers toward --mining-min-gas-price, RSKj's miner.minGasPrice, 0 unless set, where rustock kept the parent's when it wasn't set. The config file's [mining.stable_gas_price], RSKj's miner.stableGasPrice with RSKj's settings, can hold it stable in a currency instead: a configured price over the price of a bitcoin, which a web API answers as JSON or a contract answers to a call on the node, with the fixed price standing in until one is had. It has no flag, as RSKj's has none.
  • An RPC method that panics answers -32603 "Internal server error", as RSKj answers an exception, where the connection used to drop. eth_call and eth_estimateGas, which run on a blocking thread, keep upstream's -32603 "call failed".

For the code owners

  • Screening, by the three questions in CLAUDE.md:
    • Cargo.lock gains no package.
    • Nothing that runs at build time is added or changed: no build.rs, .cargo/config.toml, rust-toolchain.toml or proc-macro dependency.
    • One non-Rust executable is changed: .github/workflows/ci.yml, which gains the macOS and Windows job below.
  • Dependencies: no new crate enters Cargo.lock. num-bigint 0.4.6, already in it, becomes a direct dependency of core and execution, for the values RSKj holds as BigInteger (a gas limit or minimum gas price of any size). core's tests use sha2's compress feature, to check the merged-mining SHA-256 resumed from a coinbase's midstate. flatdb takes rustix on Unix only; Windows reserves a segment with set_len. For --import, rustock-cli now depends on serde_json, until now one of its dev-dependencies, and on sha2, beside the tokio-rustls (with ring and tls12 only) and webpki-roots that upstream's external-IP lookup brought, and deny.toml's note on webpki-roots' licence names every use it now has.
  • CI: .github/workflows/ci.yml gains a job that builds and runs the whole suite on macOS and Windows.

For review

  • 750cdfd replaces the Bridge's partial merkle tree with bitcoinj-thin's, ported into core, and removes 23 of the Bridge's 26 tests of the old tree. The new tree's tests port RSKj's MerkleBranchTest and MerkleTreeUtilsTest vectors and each of bitcoinj's refusals.
  • crates/cli/src/https.rs, the web client --import brought, also fetches the stable minimum gas price's web source, as Java's HttpURLConnection fetches it for RSKj's SimpleHttpClient: over http:// as well, any 2xx an answer, and Java's redirects, resolved as java.net.URL resolves them; --import fetches as it did. The answer is read as RSKj's Jackson 2.15 reads it (crates/cli/src/stable_gas_price/java.rs). Two limits RSKj doesn't have: an answer past 1 MiB is no price, and a redirect to a proxy the server names (305) isn't followed.
  • Not in RSKj's release, so not here either: cardamom1000, RSKj's next upgrade, which is unscheduled on mainnet and testnet and in no release (rustock takes the name and does nothing with it, as its docs say); and a preset for testnet2 (chain 34), which RSKj's master adds but hasn't launched.

A note on RSKj's miner

RSKj's miner reads a version-2 header's base event (RSKIP-535) from its trie store at the new state root, which executeAndFill never saves, so it logs "Failed to set baseEvent in block header" and writes the field empty, unless the block left the state as it was. No rule checks the field, and testnet's Union Bridge holds no base event, so both clients write it empty today.

How it was checked

  • The replay of both networks, above.
  • RSKj itself, on its regtest: 38 cases, each built as real transactions on RSKj's release and imported by the replay's build with every state root checked against its header, agree with RSKj block for block and answer for answer, and a build from before each fix parts from RSKj where the fix says it would.
  • RSKj's own tests as the checklist: the 1,239 that test these rules each have a verdict in docs/rskj-tests/, and none of the 248 in docs/rskj-test-parity.md, of its validators, its miner and its pool, is left a gap.
  • Each of the 77 switches by which rustock reads an upgrade, turned round one at a time, makes a test fail.
  • Every test passes: 4,204 on Windows, 4,205 on Linux and the smtp build's 170, with clippy clean and warnings as errors on both, and the whole suite on macOS and Windows in CI: the fork's run, where all six jobs pass, on this branch with one more commit that lets a push to a ci/ branch start the workflow.
  • Each fix comes with a test that fails without it.

How the speed was measured

On one Windows 11 laptop with an Intel 24-core processor, under Linux in WSL2, on 17 November, with nothing else of note running. RSKj's release jar VETIVER-9.0.4, its hash checked against the signed SHA256SUMS, on Java 21 with -Xmx8g and its log file at INFO rather than its shipped DEBUG. rustock from this PR at 8a00981, in a release build. For catching up, loading and the time per block, each client was started with --import on an empty data directory, every other setting at its default but the ports, rustock and then RSKj: both loaded Rootstock's copy of #9,414,000 and caught up to #9,420,907, which is the same on both and on RSKj's public node (hash 0x44290730…, state root 0x8e7d26f2…). The re-runs read each client's own database holding #9,405,000 to #9,412,300: RSKj's from its --import of Rootstock's copy at #9,405,000 on 14 November, and rustock's from the same copy, caught up from mainnet peers by this build. Block #9,412,300 is the same on both and on RSKj's public node (hash 0xb96a7a15…, state root 0x681e1c78…).

rskj's DifficultyUtils.difficultyToTarget divides 2^256 by the
difficulty, in BigInteger, and by 3 where the difficulty is 2 or less.
rustock divided 2^256 - 1 and refused a difficulty of zero, which rskj
judges as 3. At a difficulty of 1 every hash passed, and at 2 half of
them, where rskj passes one in three; and where the difficulty is a
power of two the target came out one below rskj's. On mainnet, whose
difficulty is never under 7 x 10^15, only that last difference remains.

core's merged_mining::difficulty_to_target is now rskj's in 256 bits:
(2^256 - 1) / d, one more where d divides 2^256, never over less than 3.
MergedMiningRule judges by it, and the miner re-exports it rather than
keeping its own, so the target mnr_getWork hands out is the one a
submission is judged by. DifficultyZero is gone.

Test helpers that sealed at difficulty 1 with any nonce now mine for one,
as does the load test. The miner's target test names the validator's
function, not its own re-export, and compares the two at rskj's corners.
docs/merged-mining.md no longer says the target is U256::MAX / d.

Nine mutants are all caught: the old U256::MAX / d, a floor of 2 or 4,
no extra one for powers of two or always one, a rule with a target of
its own or refusing zero, and a miner keeping the old function or one
without the power-of-two step.
Freezer::open_read_only opens a freezer to read beside the process that
writes it. It creates nothing and repairs nothing: open truncates a data
file past what the index accounts for, which in another process's
freezer is that writer's append in progress. It refuses every write
before a file is touched, since rolling to a new data file opens one to
create and truncate, and read-only handles alone would not stop that.

verify_stored_headers attaches it, so a snapshot-synced node, which
indexes by number only the window it fetched bodies for, answers every
height. It now also checks fork-detection data as BlockProcessor does,
against the 449 blocks below each block: each thread holds the headers
of its walk until those are in hand, and reads on past its range.

On a mainnet store synced from genesis to Rustock-dev#9,392,253, every header
passes the trunk rules, with none out of order, orphaned or missing, and
fork-detection data matches on all 7,801,254 blocks from wasabi100. As
controls, on the same store at Rustock-dev#9,381,160, a halved target failed
4,254,513 headers, a UMM root required one block late failed Rustock-dev#2,392,700
alone, and a zeroed uncle-count byte failed all 7,790,161.
Below RSKIP98 (orchid), rskj's ProofOfWorkRule takes a block with no
coinbase and no merkle proof, at no more than the network's
fallback-mining difficulty, as fallback-mined: its Bitcoin-header field
holds an RLP list [v, r, s], a signature of the header's merged-mining
hash by one of two fixed keys, the first at even heights and the second
at odd ones. rustock skipped proof of work below orchid altogether, so
such a block passed with any signature or none.

MergedMiningRule now asks rskj's question first and checks the
signature as validFallbackBlockSignature does: by rskj's lenient RLP
(rskj_rlp, a port of RLP.decode2), exactly one list of exactly three, a
one-byte v of 27 to 30 read as a Java byte, r and s exactly as
BigInteger.toByteArray writes them, s below half the curve order, and
the key recovered by libsecp256k1, rskj's default native library.
ChainConfig carries the fallback-mining difficulty per network, as
rskj's Constants do: 14 x 10^15 on mainnet and testnet, zero on
regtest. Merged mining below orchid is still unchecked; its partial
merkle tree comes next.

Every mainnet header to Rustock-dev#9,392,253 passes, 165,328 of them
fallback-mined, Rustock-dev#1 to Rustock-dev#365,411; verify_stored_headers now counts them.
As controls, swapping the two keys fails all 165,328, and so does
checking the signature against the block hash instead of the
merged-mining hash. The tests take mainnet Rustock-dev#1 and Rustock-dev#2 as stored, and
signatures made with k256 for each way rskj refuses one, and every one
of 32 mutants of the check, its RLP and the per-network difficulty is
caught.
rskj reads a Bitcoin partial merkle tree with bitcoinj-thin's
PartialMerkleTree (v0.14.4-rsk-20) wherever it has to show that a
transaction is in a Bitcoin block: below RSKIP92, a merge-mined block's
coinbase, and in the Bridge, peg-ins, flyover peg-ins and registered
coinbases. rustock's Bridge had a tree of its own that lacked bitcoinj's
refusals: more than 16,666 transactions, fewer flag bits than hashes,
two nodes that form a 64-byte transaction, and counts read as Java ints.

rustock_core::partial_merkle_tree ports bitcoinj's parse and
getTxnHashAndMerkleRoot, in display order as bitcoinj holds hashes, and
rskj's hasExpectedSize in Java's exact int arithmetic. The Bridge's pmt
module is now an adapter over it that gives its callers wire order, as
before. The proof of work below RSKIP92 will use the same tree.

MerkleBranch is now rskj's: display order, at most 32 hashes, no more
significant path bits than hashes (none for a negative path), the path
as BigInteger.intValue reads it, and reduceFrom refusing two nodes that
form a 64-byte transaction, as MerkleTreeUtils.combineLeftRight does.

That fixes getBtcTransactionConfirmations. It climbed the branch in
wire order and compared the root with the registered witness merkle
root, which is stored in display order, so a transaction proved by the
witness tree got -5 where rskj counts its confirmations. Now all of it
is in display order, and a branch through a 64-byte transaction fails
the call as rskj's throw does. The comments that called the ABI's
witness root internal order now say display order.

The tests port rskj's MerkleBranchTest and MerkleTreeUtilsTest vectors,
walk mainnet's first peg-in (Rustock-dev#6,916) to its block's root, take each of
bitcoinj's refusals, and on regtest register a coinbase's witness root
and ask for confirmations through both trees, in both byte orders, and
through a 64-byte transaction. 34 of 36 mutants are caught; the other
two make checked additions in hasExpectedSize unchecked where they
cannot overflow, and change no result.
Below orchid, rskj's ProofOfWorkRule checks a merge-mined block as it
does above it but for two things, and rustock skipped the check there.

The Bitcoin-header field. From RSKIP98 it must be exactly a Bitcoin
header. Below it, and at block 0 always, BtcHeaderSizeRule passes any
size and bitcoinj-thin's makeBlock reads the field: a header, then, if
more bytes follow, the block's transactions, throwing where they do
not parse. rustock_core::btc_block reads it as bitcoinj does, with
Java's int and long counts, a negative list size refused as ArrayList
refuses it, and the 32 MiB cap on a read. Only whether it throws
matters, since the proof of work is the first 80 bytes' either way.

The merkle proof. Below RSKIP92 it is a bitcoinj partial merkle tree,
which GenesisMerkleProofValidator requires to be exactly the size its
counts declare, to walk to the Bitcoin header's merkle root, and to
match the coinbase, both compared in display order. It is the
bitcoinj-thin tree in core, the one the Bridge reads.

Every mainnet header to Rustock-dev#9,392,253 passes. Of the 728,999 below orchid,
165,328 are fallback-mined, their signatures already checked, and
563,671 merge-mined, their trees checked now; verify_stored_headers
counts them. As controls, comparing the header's root in wire order,
matching the coinbase in wire order, and refusing a field that holds
the header alone each fail exactly those 563,671, so every one of them
carries a bare header. The tests take mainnet #728,999, the last block
below orchid, as stored, and bitcoinj's reading of the Bitcoin genesis
block and of each count it reads as Java does; all 25 mutants of the
new code are caught.

The docs said the rule skipped blocks below orchid. They now say what
it checks there, and the map names the files.
decode_with_hash cached the keccak of a header's bytes as sent. rskj
hashes a compressed form instead, and for a versioned (RSKIP-351) header
that form is not the wire form: the wire carries the literal bloom in
slot 6 and the version, edges and base event after ummRoot, while the
hashed form leaves those three out and puts a commitment to all four in
slot 6. Every testnet header from reed810 (Rustock-dev#7,139,600) on got a block
hash and a merged-mining hash no other node agrees with, so no header
named its parent, and the testnet snapshot sync refused every chain it
was offered as not forming one.

Both hashes are now built from the items as the peer spelled them, less
the version fields and, from RSKIP-92 on, the merged-mining proof and
coinbase, with the commitment in slot 6. A version-0 header hashes
exactly as before.

decode_with_hash_and_activations does the same under explicit
activation heights, as decode_with_activations does: RSKIP-144's height
decides what a lone item after a version-2 byte is. decode_with_hash
keeps reading under the process-wide heights.

The tests check the network's decoder on three captured testnet headers
(version 1, version 2, and version 2's short form), read under testnet's
heights, and on Rustock-dev#8,160,760, against rskj's block hash and the
merged-mining hash each miner committed to in its coinbase, and on every
layout the decoder reads, a version-2 base event without RSKIP-144
included, against the hashes this node derives by encoding. 18 of 19
mutants of the new code are caught; the survivor, rlp_list_items'
list-length check, cannot fail at its one call site.
From RSKIP290 on, rskj holds testnet's difficulty to at least
550,000,000 (Constants.getMinimumDifficulty), and testnet.conf activates
it at Rustock-dev#2,581,800 rather than at hop400. rustock held testnet to 131,072
at every height, so at Rustock-dev#6,360,768, a late block whose parent sat at the
floor, it expected 548,625,000 where the chain has 550,000,000, and the
testnet snapshot sync refused the header after walking 1.9 million
others.

The floor in force at a height is ChainConfig::min_difficulty_at, which
both of the difficulty rule's floors now use: the clamp on a falling
difficulty, and the ten-minute drop that testnet never retires. rskj
applies RSKIP290 on testnet alone; mainnet's floor stays 7e15, which
taking it there would lower.

The tests take Rustock-dev#6,360,768 and its parent from the live chain, the
activation height, the ten-minute drop on either side of it, and
mainnet and regtest, which keep their floors. Each of 8 deliberate bugs
fails a test.
…walks

A relay answers a light client's Rootstock requests from its own node, and
the client checks every byte, so the node gives the bytes as rskj writes
them:

- rsk_getBlockNumberByHash(hash): the block's number on the chain the node
  follows, or null, as for a block held only on a fork.
- rsk_getRawBlocks(from, to): one run of that chain, at most 1,000 blocks,
  oldest first, each its header compressed, as rskj sends a chain's headers
  to its peers (BlockHeadersResponseMessage, getEncodedCompressed), and its
  uncles in full, the form its uncles hash covers. It stops before the first
  block it doesn't hold whole and before one that doesn't follow the one
  before it. Uncles come from the block's body or, once that's gone, from the
  freezer; a block whose uncles hash is the empty list's is whole without
  either; an uncle held only compressed isn't held whole.
- rsk_getStateNodes(blockHash, [keys]): for each raw unitrie key, the nodes
  of the walk along it from the block's own state root, deepest first, as
  try_nodes_toward gives them; null if the node doesn't hold that state, and
  an error if the store can't produce a node on the way. At most 64 keys, of
  at most 128 bytes each.

Header::encode_compressed writes a header compressed. Testnet Rustock-dev#8,260,761,
which the testnet node took in full, compresses to what a converter written
apart from this crate makes of it, with rskj's hash; Rustock-dev#8,260,720, which rskj's
peers sent compressed, writes back as sent. The README lists these methods
with rsk_getStorageNodes. Each of 24 deliberate bugs fails a test.
A state node longer than rskj's 51,200-byte chunk window is never one
of a chunk's own nodes, wherever it is asked for: rskj keeps a node
only if it ends inside the window, and leaves the first that doesn't on
its stack, which it hands back top-first, the node whole. The client
counted a chunk by its own nodes alone, so it never counted such a node.
Asked at the cursor, rskj's walk began with the node that ends there;
when that node has an embedded right child and no separate left one,
the walk counts it from inside the child, never keeps a long one, and
stops at it at once. Every answer was taken, the cursor didn't move, and
each answer taken reset the count of fruitless ones: on testnet at
Rustock-dev#8,261,160 the download stood at 94% for hours.

- The trie's rebuild places the node the walk stopped at
  (Rebuilt::stopped_at), and with none of the chunk's own, gives that
  node the first node's left hash: without it an account over its
  storage, with its code embedded in it, doesn't rebuild.
- A chunk covers its own nodes and the node it stopped at, and the next
  request asks just past the cursor. What a slice asks next is one
  value, Ask: the cell, the cursor, or just past it.
- Each slice keeps the offset it asked. An answer used to be matched by
  the cursor or the cell holding it, so a slice asking exactly took the
  answer of a slice that had asked for that cell, and the first answer's
  reshaping of the slices could leave a request no slice would match.
- An answer taken that moves nothing counts as fruitless; 64 in a row
  end the session, as 64 refusals do, blaming no peer.

Tests download from rskj servers through nodes longer than the window,
tile three shapes of trie asking just past where each chunk stopped,
place long nodes over a separate left child, give up on answers that
move nothing, and take an answer to the slice that asked for it.
docs/snapshot-sync.md says where a chunk stops. Each of 20 deliberate
bugs fails a test.
From RSKIP-351 rskj sends a chain's headers to its peers compressed: a
version-1 or version-2 header holds, in its bloom's slot, only
RLP([version, keccak(extension)]), and the extension -- the block's logs
bloom, execution edges and base event -- comes beside the block's body,
as the third element of BodyResponseMessage.
DownloadingBodiesSyncState.newBody sets it on the header before the
block is built and run. rustock's decoder dropped it, so a block whose
header came compressed ran against an empty bloom: after its snapshot
the testnet node executed Rustock-dev#8,261,161 and stopped at Rustock-dev#8,261,162 on "logs
bloom mismatch", its receipts root matching. Not even an empty block's
bloom is empty, since REMASC logs each fee it pays.

- A BodyResponse carries the extension, framed as rskj frames it: a body
  of exactly three elements has one, read whole as
  BlockHeaderExtension.fromEncoded reads it, and a third that isn't one
  is refused; a body of any other length has none.
- Header::with_extension completes a header that holds only the
  commitment with the extension it names, once accepts_extension holds
  and the bloom is 256 bytes long. The hash stays the same, and the
  header keeps its own merged-mining fields. Header::extension gives the
  extension a header holds, and none for one that holds only the
  commitment.
- The sync takes a body for such a header only with that extension
  (body_check::completed_header). The completed header is stored before
  the body, and the batch, or follow mode's buffer, runs it. A body
  without the extension, or with another, is refused as a body that
  doesn't match its header is, and asked of another peer.
- Such a header's body is neither derived from the header nor filled
  from a local block source, which has no extension to give. One already
  stored beside it -- derived before this -- is dropped when its batch
  starts or follow mode reaches it, and asked for again.
- A served body carries its header's extension, as
  NodeBlockProcessor.processBodyRequest sends it. A header holding only
  the commitment has none to send, so its body isn't served.
- BlockStore::delete_body.

rskj sets whatever extension comes and lets execution find out; this
takes only the one the block's hash covers. The blocks a snapshot
fetches below its checkpoint still keep the walk's compressed headers.

Tests complete testnet Rustock-dev#8,260,761 and versioned headers of both versions
from their compressed form, byte for byte and hash for hash, and refuse
another bloom, another version and a short bloom; frame bodies with and
without an extension against rskj's framing; and take, refuse and ask
again for bodies in a batch and in follow mode, drop stale bodies, and
serve the extension. Each of 29 deliberate bugs fails a test.
rskip-spec-vs-rskj-code.md said the compressed form of a versioned
header exists only to be hashed and is never sent. It is sent:
BlockHeadersResponseMessage gives a chain's headers to peers with
getEncodedCompressed, and their bloom, edges and base event come beside
each block's body, as BodyResponseMessage's third element. Whole blocks,
in BlockMessage and in a snapshot's blocks, carry the full form. Both
tables now say which form goes where, and the second says how rustock
takes the extension.
rsk_shutdown's doc gave the operating procedure as "SIGTERM and wait,
never SIGKILL", as if SIGTERM stopped the node cleanly. Only an rskj
import installs signal handlers; the node installs none, so SIGTERM ends
it as abruptly as SIGKILL. That is safe, since the executed head never
advances past what a crash keeps -- the testnet node, stopped with
SIGTERM on 7 November, resumed from its executed head -- but it is
rsk_shutdown that flushes the trie store first and leaves nothing to
execute again.
rskj recovers a transaction's sender with libsecp256k1 by default, and
when libsecp256k1 refuses to recover a key, asks whether the key would
have been the point at infinity: if so, it answers the key whose
encoding is the one byte 00 (Secp256k1ServiceNative.recoverFromSignature,
"to be compatible with BC implementation": Bouncy Castle encodes the
point at infinity so). ECKey.getAddress hashes that encoding past its
first byte, which is nothing, so the sender is keccak256("")[12..],
0xdcc703c0e500b653ca82273b7bfad8045d85a470. rustock refused such a
signature, so the testnet node stopped at Rustock-dev#8,269,753: the first
transaction in Rustock-dev#8,269,754, a transfer from that account at nonce 10,
is signed so that sR = eG.

- recover_sender and btc_sender_hash160 take the key from one place,
  recover_key, which answers the point at infinity where libsecp256k1
  refuses a signature whose R exists, whose r and s aren't zero, and
  for which sR = eG, e being the signing hash modulo the order. Any
  other signature libsecp256k1 refuses is refused as before.
- Its sender is INFINITY_SENDER. A peg-out from it pays the hash160 of
  the one byte 00, as rskj's
  BridgeUtils.recoverBtcAddressFromEthTransaction does from that same
  encoding; recover_compressed_pubkey has no 33-byte key for it and
  answers none.

Execution, the pool, mining, alerts and the RPC all recover senders
through recover_sender. The ECRecover precompile is revm's, which
answers nothing for such a signature where rskj's answers this sender;
that is left for a change of its own.

Tests recover testnet's transaction, byte for byte and hash for hash,
as rskj does, with its peg-out address; check the sender is the hash
of nothing; recover the point at infinity from a signature made so
that sR = eG, and an ordinary key from the same r with y's other
parity; and still refuse an R off the curve and a zero s.
The tests of rsk_getStorageNodes checked that its nodes chain up to the
block's state root, through this node's own trie store. A light client has
neither that store nor any reason to trust the node: it has a block hash it
trusts, and the bytes it is sent.

read_slot_as_a_light_client does what such a client does: it files each
node under its own hash in an empty store, requires the top node to hash to
the header's state root, and walks the slot's key with try_get, so a node
left out of the answer fails the walk instead of reading as an absent slot.

The test reads a contract's storage that way at two blocks, the second
rewriting one slot, clearing one and writing a third, each from a raw
header checked against the hash the client trusts. An answer with an
interior node removed is refused as incomplete, and the earlier block's
answer is refused against the later block's state root.
thread_cpu_time read CLOCK_THREAD_CPUTIME_ID on Linux and returned zero
everywhere else, so on macOS and Windows the chain-activity report's CPU
figures, cost per gas among them, were always zero, and
thread_cpu_time_advances_while_working failed there.

macOS has the same clock from 10.12. Windows has none: GetThreadTimes
gives the thread's kernel and user time in 100 ns units, advancing by
scheduler ticks of about 15.6 ms, which the report's averages over many
blocks absorb. Other platforms still report zero.
rustock-flatdb did not compile on Windows: it locked the database
directory with rustix's flock and reserved each segment with rustix's
fallocate, and rustix has neither there.

The lock is std's File::try_lock: flock on Unix, as before, and
LockFileEx on Windows, and a lock held elsewhere still reads as
WouldBlock. reserve keeps fallocate on Unix; on Windows it is set_len,
the fallback Unix already takes where a filesystem refuses fallocate, so
rustix becomes a Unix-only dependency.
read_env_file refuses an env file that group or others can read, by its
Unix mode bits, and on Windows the metadata it read for them went unused,
which -D warnings made an error, as was the test helper's unused mode.

The metadata is read inside the Unix check, so Windows reads only the
file. Windows keeps access in ACLs, which the check does not read, so
there the file is not checked, as before, and the test of the refusal
runs on Unix only.
main ran the node on the process's main thread, whose stack is the
OS's: 8 MiB on Linux and macOS, 1 MiB on Windows. A debug build
overflows 1 MiB while clap builds the parser for the CLI's 136 options,
so on Windows even rustock-cli --version crashed, and so did every test
that runs the binary (the four in tests/import_blocks.rs).

The node now runs on a thread with an 8 MiB stack, the stack Linux and
macOS already gave it, set by the node rather than by the OS. The local
offset is still read before any thread is spawned, a panic on the thread
still ends the process with it, and the thread is named main, so logs and
panic messages read as before.
An upgrade ladder says when a rule is on only on a network that runs
every rung of it. Two networks skip one: mainnet never schedules reed810
and testnet never schedules arrowhead631, and on each the next upgrade
outranks the skipped one, so `active_upgrade() >= X` reads as true there
for good. And rskj's network files move some rules off their upgrade
altogether. One Bridge gate and three predicates were wrong:

- The union bridge's seven methods (RSKIP502, reed810) were gated
  `Since(Reed810)`, so on mainnet they became methods at Rustock-dev#8,804,200,
  when vetiver900 arrived. rskj never enables them there: its parseData
  finds no method, so a call to one fails (a direct transaction paying
  its whole gas limit) while rustock executed it. One such call on
  mainnet would have split rustock off the chain. They now follow
  has_rskip502: reed810 scheduled, which is testnet from Rustock-dev#7,139,600.
- RSKIP284 is at Rustock-dev#2,581,800 on testnet (testnet.conf), not at hop400
  (Rustock-dev#3,103,000). Besides the BTC address checks, it moves testnet's
  federation UTXOs into the pre-hop cell `newFederationBtcUTXOsForTestnet`
  for the 521,200 blocks before RSKIP293. rustock had the cell but, with
  RSKIP284 at hop400, never reached it. RSKj's public testnet node shows
  the cell empty through Rustock-dev#2,581,800 and written at Rustock-dev#2,581,801. This
  matters for a testnet sync from genesis.
- RSKIP434 asked `>= Arrowhead631`, true on testnet from lovell700, where
  rskj never activates it. Inert, since the guard it lifts applies only
  to Bitcoin mainnet. `all_active` now names arrowhead631, so tests keep
  it on.
- RSKIP535 was on from vetiver900 on mainnet. reference.conf has it at -1
  and main.conf never sets it; testnet.conf sets 7,604,200 and
  regtest.conf 1. Nothing in consensus reads it yet.

A new test holds 68 predicates to rskj 9.1's schedule on both networks:
each must turn on exactly at rskj's height or, where rskj says -1, at
none of the heights either network schedules anything at. has_rskip502
and has_rskip529 (RSKIP502's schedule) are added for the Bridge.

rustock's regtest does not schedule reed810, as rskj's regtest.conf
does, so the union bridge's methods are now off there, as RSKIP144 and
RSKIP351 already were.
rskj's BridgeMethods marks getUnionBridgeContractAddress and
getUnionBridgeLockingCap ALLOW_STATIC_CALL, like every other read-only
getter. rustock's list of static-callable methods left them out, so a
contract that STATICCALLed either one had the call refused here
(validateCallMessageType's check, which spends the method's cost and
returns nothing) while rskj answered it. The contract then went on from
a failed call where the chain went on from a value. Testnet only, from
reed810.

A test now pins the count: of this table's methods, rskj marks 52
ALLOW_STATIC_CALL, counted from its enum.
rskj 9.1's Bridge has 83 methods and rustock had 77. The six missing are
RSKIP529's, on reed810's schedule (testnet from Rustock-dev#7,139,600, never on
mainnet): getSuperEvent, setSuperEvent, clearSuperEvent, getBaseEvent,
setBaseEvent and clearBaseEvent. A Bridge method rustock does not know
fails as a malformed call, so the first testnet transaction to use one
would have parted rustock's state from RSKj's and stopped the node.

As rskj's UnionBridgeSupportImpl and UnionBridgeStorageProviderImpl have
it:
- Each event is a byte string in the keccak-keyed cell "superEvent" or
  "baseEvent", stored as it came, with no RLP around it. An absent cell
  reads as the empty array, which the getter returns ABI-encoded in 96
  bytes (BytesType pads empty data to one word).
- Setting and clearing are for the union bridge contract only, checked
  against the immediate caller as the RSKIP502 methods are. Any other
  caller, or an event over 128 bytes, is an IllegalArgumentException
  rather than a response code, so the call fails as any Bridge method
  that throws does: its cost is spent and nothing is written.
- Clearing, or setting the empty array, deletes the cell.
- The getters cost 3,000 and take a STATICCALL; the rest cost 8,000.

The bytes argument is read the way rskj reads it, each offset and length
word as a Java int, so high bits set in either are ignored rather than
overflowing a conversion to usize.

Under RSKIP535, rskj also copies baseEvent into the header of a block it
mines (BlockExecutor.setBaseEvent). rustock's miner does not build
RSKIP351 headers yet, so that is left to the miner's own change.
…n its order

A new bridge::abi decoder follows rskj's CallTransaction.Function.decode
and SolidityType exactly, and every Bridge method now reads its arguments
through it:

- an offset, length or count is its word's low 32 bits as a Java int
  (intValue), so high bits are ignored and a negative one throws, where
  converting the whole word with U256::to panicked;
- integers are signed, an empty payload reads as zero for the integer
  types (int, uint, bool, address) and fails for the others;
- an address goes through DataWord, which drops a negative word's
  leading 0xFF bytes; a bool is intValue() != 0; a string is lossy UTF-8;
- getGasForData parses the call as execute does, so arguments that do not
  decode cost the flat 23,000 there too.

What changes with it, each as rskj answers:

- getBtcBlockchainBlockHeaderByHash answers 96 bytes for an unknown
  block, and ...ByHeight uses the full main-chain lookup;
- receiveHeaders refuses the whole call when any header has the wrong
  size;
- lock whitelist: the address is checked against the network, in rskj's
  order of checks; addOneOff's value and the disable delay are read with
  longValueExact / intValueExact, which throw; the delay sum is Java int
  arithmetic; voteFeePerKbChange answers GENERIC_ERROR for a fee that is
  not a long;
- isBtcTxHashAlreadyProcessed and getBtcTxHashProcessedHeight throw for a
  hash string that is not 64 lower-case hex digits;
- negative indexes find nothing, and an empty index is index 0;
- addSignature refuses a key bitcoinj cannot parse, an empty signature
  list, a non-DER signature and a hash that is not 32 bytes: a success
  with no output before RSKIP88, a throw from it; the point at infinity
  parses;
- the union bridge's amounts and caps are signed, so a negative one is
  INVALID_VALUE instead of wrapping the running total;
- releaseBtc and getEstimatedFeesForPegOutAmount take the satoshis'
  longValue, as Coin.toBitcoin does.

registerBtcTransaction and registerFlyoverBtcTransaction now share one
validationsForRegisterBtcTransaction, in rskj's order: height and
confirmations, the tree's size, the merkle root, validateInputsCount (new:
a transaction without inputs throws), the block lookup with RSKIP199's
search limit (it throws below it, and a block the store does not have
throws, where both were swallowed), then the merkle root against the
block. The flyover answers a false with the validations code and a throw,
or a transaction that does not parse, with GENERIC_ERROR, as
Bridge.registerFlyoverBtcTransaction's catch does.

core's bitcoinj VarInt reader is public for validateInputsCount.

Tests: the decoder's rules; offsets, counts and heights with 2^64 added
through the executor for receiveHeader, receiveHeaders,
getBtcTransactionConfirmations and registerFastBridgeBtcTransaction; the
validations' order and answers; a crafted input-free peg-in failing the
transaction; the flyover's codes; addSignature before and after RSKIP88;
the union, whitelist, hash-string, index and header changes; and every
method-table signature decoding.
…on's own call is the invisible exception

HDWalletUtils, BlockHeader and Environment are rskj's native contracts
(co.rsk.pcc.NativeContract) and now run as one: a new native module reads
a call's selector and arguments with the shared crate::abi decoder (moved
from bridge::abi), charges the method's getGasForData before it runs, and
settles a throw as rskj does. Nothing they read from a call can make them
panic: offsets, counts and lengths are rskj's Java ints.

A precompile that throws in a transaction's own call is rskj's invisible
exception (TransactionExecutor.call), now for every precompile, not only
the Bridge and REMASC: the receipt is a success with no output that used
the intrinsic gas and getGasForData, the sender pays for the whole gas
limit, REMASC receives it, the value stays with the sender, and the
precompile's account is not created. The precompile's frame reverts, so
revm undoes the value and the touch, and RskHandler's last_frame_result
turns the result back into a success. In a contract's call a throw costs
getGasForData and the call fails from RSKIP197; before it, it spends the
call's gas (Rustock-dev#295). A call short of getGasForData is out of gas before the
precompile runs.

What changes with it, each as rskj answers:

- the natives are precompiles only from their activation; Environment and
  BlockHeader used to answer at their addresses before it;
- Environment's getCallStackDepth is the caller's depth plus one, the
  contract a transaction calls being depth 0 (it answered one too many);
- BlockHeader walks up from the executing block's parent by hash until it
  meets the main chain, as getBlockAtDepthStartingAt does, so a block on a
  fork reads its own ancestors, and past depth 0 the genesis block is out
  of reach; 1 to 31 bytes after the selector do not parse;
  getMergedMiningTags throws for a block without a coinbase (an empty RLP
  field is rskj's null) and for a tag whose hash runs past the end;
  getBitcoinHeader answers no bytes at all for a block without one; an
  uncle index is read before the uncles, which come from the body, the
  freezer or the header's uncle count;
- a block, body or total difficulty the store lacks, which rskj always
  holds, stops the node instead of answering empty. The header sync
  records a total only for each run's newest block, so for
  getCumulativeWork that gap is still open;
- HDWalletUtils follows bitcoinj: the xpub and tpub prefixes and headers,
  a depth that wraps past 255, CKDpub through libsecp256k1, a path read as
  Java's Integer.parseInt reads it (signs, leading zeros, decimal digits of
  any script), hybrid keys in a multisig;
- BN128 refusing a point answers no bytes before RSKIP197
  (BN128PrecompiledContract.safeExecute) and throws from it; a modexp
  length no int holds is out of gas, as rskj prices it above 10^16.

Tests: rskj's vectors for every HDWalletUtils method and a payload sweep
that never panics; BlockHeader's depths, forks, genesis, uncles, RSKIP536
and a store with gaps; Environment three calls deep
(nested_environment_calls); the invisible exception's receipt, fees, value
and account; out of gas before a native runs; BN128 either side of
RSKIP197; and a contract's throwing call either side of RSKIP197.

The old implementations' tests in precompiles.rs go with them; every
case they checked is in the new tests, with rskj's own test constants
where those differ.
…ocks, RPC, precompiles

Everything a stranger can send was tried, and what it broke is fixed.

Peers. A panic in a session now ends that session and drops its peer,
and nothing else: register_and_run_session runs the session under
catch_unwind, where nothing caught a panic outside peer discovery.
Panics a peer could cause:

- discovery filed a node in one of 256 buckets by its 512-bit XOR
  distance, so a node ID whose high bits differ from ours indexed past
  the end. Nodes are now ordered as rskj's Kademlia table orders them,
  by the distance between keccak256(keccak256(id)) keys
  (DistanceCalculator), with the bucket read by get_mut;
- the RLPx responder read the initiator's auth signature as 65 bytes
  without checking its length;
- a frame header's long-form list length, any 8 bytes, overflowed when
  added to its start.

And two that need no peer: `Instant::now() - d` panics on a machine
booted less than d ago, which the outbound dialer's starved-warning
timer and the snapshot session's phase report both did.

Snapshot sync. A peer offering a state could make the node hang, abort,
panic in a debug build, or grow its memory without limit:

- checkpoint::max_work_between, the most work a span of blocks can
  hold, walked block by block over a span the peer chose; in a release
  build a span of u64::MAX looped for good. It now keeps only the
  ceilings that differ and counts a settled run at once, and gives what
  the walk gave (checked against it over a grid of divisors, spans and
  difficulties);
- the sampler's fill_gaps placed its points one at a time over a gap the
  peer chose; it now skips to each next point, with the same result;
- a skeleton answer is held to what rskj sends
  (NodeBlockProcessor.processSkeletonRequest): at most 21 identifiers,
  all within 3,840 blocks of the grid point at or below the start;
- the offered header's proof of work is checked only against its own
  difficulty, so a state more than 10,000,000 blocks above the
  checkpoint, about 9.5 years of mainnet, is refused
  (SnapFailure::ImplausibleHeight), and the peer blamed as for an
  invalid block.

The snap client measures a slice's remaining length with saturating_sub,
as its rebuilt-chunk path already did.

Blocks and transactions. A transaction's v overflowed while its
recovery id was computed, and a receipt with a bloom over 256 bytes
panicked.

Locks. The 42 places that took a std lock with unwrap or expect
panicked if another thread had panicked holding it; they take it with
PoisonError::into_inner.

RPC. A method that panics is answered as rskj's RskErrorResolver
answers an exception it has no answer for, -32603 "Internal server
error", where the client's connection used to drop with every other
request in its batch. trace_filter and eth_getFilterLogs walked every
number up to a toBlock as high as 2^64, a request that never ends; both
stop at the head, as rskj's walk stops at the first block it doesn't
find. eth_getLogs' span check overflowed for a fromBlock near 2^64.

Tests. core::mutate varies real encodings byte by byte, and fuzz tests
now cover: headers of every version with their merged-mining fields and
extensions; transactions and receipts; rskj RLP; Bitcoin blocks and
PMTs; RSKj's snapshot chunks, rebuilt against their roots; every RSK and
P2P message, varied from real encodings and with random RLP for every
type and id; the Bridge reading a peg-in's scripts, witnesses and
outputs (sender shapes, peg-in instructions, federation spends); and
every precompile, each method with ABI edge words, called directly, by
CALL, STATICCALL and DELEGATECALL and as a local call, by a stranger and
by the federators and authorizers, under genesis rules, mainnet's
between Papyrus and Iris, testnet's, and everything on. The range tests
run on a thread of their own with a minute to answer, so a regression
fails rather than hangs.
Until RSKIP191 (iris300: mainnet Rustock-dev#3,614,800, testnet Rustock-dev#2,060,500) rskj's
VM has three opcodes of RSK's own, and rustock had none of them: revm
has no opcode at 0xa8 to 0xaa, so a contract that used one failed and
spent all its gas where rskj ran on, and its block's state root split.

- DUPN (0xa8, 3 gas) pops a depth d and is DUP(d + 1); SWAPN (0xa9, 3
  gas) pops d and is SWAP(d + 1) (VM.doDUPN, doSWAPN). A depth the
  stack doesn't reach fails and spends all the gas, as verifyStackSize
  and DataWord.intValueCheck throw.
- Both call program.step() twice, so the byte after them never runs,
  though a jump may land on it: Program.computeJumpDests skips only
  PUSH data, as revm's analysis does. Where the second step leaves the
  code, Program.setPC stops the program. revm pads code with as little
  as one STOP and reads the next opcode unchecked, so DUPN or SWAPN
  among the code's last two bytes stops there rather than stepping off
  the end.
- TXINDEX (0xaa, 2 gas) pushes the index of the transaction in its
  block, counted from 0 as BlockExecutor counts, the same in the calls
  it makes (ProgramInvokeFactoryImpl), and 0 in a call outside a block
  (ReversibleTransactionExecutor).

From RSKIP191 all three are invalid opcodes, now installed as such
explicitly. The all-ops tracer dispatches them the same way.

Tests: DUPN and SWAPN of every depth against DUP1-16, SWAP1-16 and
rskj's stack beyond them, with their gas; depths the stack doesn't
reach, and one no int holds; the byte after them and the end of the
code; TXINDEX in a block, in a nested call and in a local call; and
all of them failing from RSKIP191.
… as in rskj

Before RSKIP197 (iris300: mainnet Rustock-dev#3,614,800, testnet Rustock-dev#2,060,500)
rskj's Program.executePrecompiled has no error handling. A precompile
that throws in a contract's call leaves the VM as a RuntimeException,
and VM.steps fails the contract that made the call with all its gas;
each frame runs under its own vm.play, so the contract above it gets 0
from its CALL and goes on, or the transaction fails with its whole
limit if the contract was its own call. rustock spent the forwarded gas
and let the caller go on. A caller that forwarded everything ran out
anyway, since RSK has no 63/64 rule, but one that held gas back carried
on where rskj stopped it (Rustock-dev#295).

- The precompile provider raises a flag where a pre-RSKIP197 throw
  reaches a contract's call (Bridge methods, REMASC, the native
  contracts and the rest), and RskHandler's exec loop halts that
  contract as the call's result reaches it.
- REMASC from a contract: from RSKIP197 its throw costs its
  getGasForData, 0, so the caller gets all the forwarded gas back; it
  spent it all before. Before RSKIP197 it fails the caller like any
  other throw.
- getStateForDebugging(), which REMASC answers for any caller, returns
  RemascState.getEncoded, ABI-encoded as rskj does, not nothing.

Tests: a capped forward to a throwing Bridge charges the whole limit
before RSKIP197 (it pinned 96,733 as a known divergence); a throw two
calls deep fails only the inner contract, which loses all the gas it was
given, and from RSKIP197 the inner one goes on; HDWalletUtils' throw
with gas held back; REMASC refusing a contract either side of
RSKIP197; getStateForDebugging from a transaction and from a contract;
the state's RLP, worked by hand from rskj's RLP. Each fails without the
change. The two Bridge helpers now pass the selector their comments
describe (argsOffset 0x1c, not 0).
…ncluded

rskj's ECRecover (PrecompiledContracts.ECRecover) is not Ethereum's,
which revm runs, in two places, and rustock ran revm's:

- A signature whose key is the point at infinity recovers the address
  rskj gives that key, keccak256("")[12..] (0xdcc703c0...a470), as
  b14c358 made rustock do for a transaction's sender. revm answered
  nothing.
- isValid reads only the last byte of the v word, so a word whose other
  bytes aren't zero still recovers a key. revm answered nothing.

Everything else already agreed, and the tests hold it: 3,000 gas; the
first 128 bytes read; nothing for a v byte other than 27 or 28, an r or
s of 0 or the curve's order or more, or a key that doesn't recover; an
input too short to hold the hash, v and r answers nothing (rskj's
System.arraycopy throws, and execute answers nothing for any exception),
and a short s ends in zeros.

rustock-core's recovery is shared: recover_address applies rskj's
validateComponents and recovers the key as recover_key does, with the
point at infinity where libsecp256k1 refuses a key that would have been
it. The precompile at 0x01 is rskj's from genesis.

Tests: signatures built so that R is the generator, with the hash 1 and
s 1 (the point at infinity) or 2 (an ordinary key), for v 27 and 28,
held against revm's ECRecover where the two should agree; v with other
bytes set; short inputs; and recover_address against a k256 signature,
the point at infinity, out-of-range r and s, and an R off the curve.
… (RSKIP516)

From reed800 (mainnet Rustock-dev#8,052,200, testnet Rustock-dev#6,835,700) rskj cuts a
contract's input to a precompile to the contract's getMaxInput
(Program.getInputLength): ECRecover, BN128 addition and secp256k1
addition 128 bytes, BN128 and secp256k1 multiplication 96, BLAKE2F 213.
The cut feeds both getGasForData and execute. A transaction's own data
reaches a precompile whole (TransactionExecutor.call), and every other
precompile takes all of it. rustock never cut it.

Only BLAKE2F answers differently. The others never read past their
limit, but BLAKE2F refuses an input that isn't exactly 213 bytes, and
prices it at 0. So from reed800 a contract that hands BLAKE2F more than
213 bytes gets the compression of the first 213 and pays their rounds,
where rustock refused the call.

has_rskip516 joins the hardfork config, and the provider applies the
cut where it reads a call's input, for a call at depth above 1.

Test: a contract passes its own input to BLAKE2F. From reed800 a
214-byte input answers as the 213 bytes before it, and costs 16 gas
more, RSKIP400's price for the extra byte of transaction data. At
Rustock-dev#8,052,199 BLAKE2F refuses it, 12 gas cheaper: 0 instead of its 12
rounds. A transaction sending the 214 bytes to BLAKE2F itself still gets
no answer. Disabling the cut, or applying it to a transaction's own
input as well, fails the test.
…s its block

rskj reads a transaction's empty data as null (RLPItem.getRLPData), and
TransactionExecutor.call asks the precompile its price before its try:
getGasForData(tx.getData()). Before RSKIP552 (vetiver900: mainnet
Rustock-dev#8,804,200, testnet Rustock-dev#7,604,200) BLAKE2F's getGasForData reads the
data's length, so a transaction that calls BLAKE2F with no data throws a
NullPointerException out of the call, and rskj can't process its block.
rustock took the call as an invisible exception and the block as valid.
No canonical block holds one.

Such a call is now Outcome::Fatal, which fails the block. RSKIP552 gives
null data a price of 0 and refuses it like any other wrong length, which
rustock already did, and that stays from vetiver900. Every other
precompile's getGasForData takes null (ECRecover, SHA-256, RIPEMD-160,
Identity, ModExp, the BN128 contracts, the secp256k1 contracts, the
native contracts, the Bridge and REMASC), and a contract's call never
hands a precompile null (Program.callToPrecompiledAddress).

Test: at mainnet Rustock-dev#8,804,199 the block fails; at Rustock-dev#8,804,200 the
transaction succeeds with no output and 21,000 gas; a contract's call
with no data at Rustock-dev#8,804,199 fails as it always did, pushing 0. Disabling
the new rule fails the test. execute_alone's body moves to
try_execute_alone, which returns what fails the block.
rskj reads a transaction (Transaction(byte[])) as exactly one RLP list
of nine elements, hashes it by an encoding it rebuilds rather than by
the bytes it was sent, and takes its signature only if
acceptTransactionSignature does. rustock read transactions leniently,
hashed the bytes as they came, and checked less of the signature, so
the two disagreed on which blocks are valid.

Reading (Transaction::from_rskj). Exactly nine elements; `to` empty or
20 bytes; v absent, for no signature, or one byte with r and s present
(ECDSASignature.fromComponents throws on none). validate()'s limits
apply: a nonce or gas limit of at most 32 bytes, a gas price or value
below 2^255, r and s of at most 32 significant bytes, and a gas limit
present, which validate() dereferences. A block's transaction may come
wrapped in a byte string, since BlockFactory hands each element's
getRLPData to the constructor. Refused too is what rskj reads but no
block or pool takes: a v naming a negative chain (below 34 but for 27
and 28) and a nonce past 64 bits.

Hashing. rskj's hash and transactions root use getEncoded, rebuilt from
the fields as it holds them: a nonce of the one byte 0 goes empty and
any other stays as it came; the gas price stays empty if it came empty
and is otherwise minimal, 0x00 for zero; the gas limit and data stay as
they came; the value is minimal; v is rebuilt from the chain id and
recovery id; r and s are minimal. That encoding is now the one kept,
hashed, put in the trie and served. For what rskj sends the two are the
same bytes, since its encoding reads back to itself; they differ for
bytes rskj never writes.

Signing. The signing hash is getEncodedRaw, from those fields and the
chain v names. extractChainIdFromV divides in Java ints, so v of 34, 35
and 36 name no chain, as 27 and 28 do; rustock refused all three.
getRealV is 28 for an even v and 27 for an odd one.

Accepting. Every transaction of a block but REMASC passes
acceptTransactionSignature (BlockTxsFieldsValidationRule): r from 1 to
below the curve order, s from 1 to below half of it, and a chain of 0
or the network's. rustock took a high-s signature, so a block carrying
one was valid here and invalid on the network.

REMASC. One predicate, Transaction::is_remasc, is rskj's
isRemascTransaction at all six places that ask: the block's last
transaction, to REMASC, with no data and no signature, and value, gas
price and gas limit zero. rustock took a REMASC-shaped transaction
anywhere in the block, and with value, data or a gas price, as REMASC.
Its sender is the zero address and is never cached: the sender cache
keeps only signed transactions, whose hash decides their sender, as
sender_cache's docs now argue.

The pool keys a transaction by rskj's hash and removes a mined one by
it. It removed by a canonical re-encoding, which missed every mined
transaction whose rskj encoding isn't canonical. Its REMASC step, which
could never fire, is gone. A transaction submitted to a node with no
pool is read, to report rskj's hash; the relay keeps keying what it has
seen by the digest of the bytes. Transaction::gas reads a gas limit as
GasCost.toGas, for the executor to take up next.

Tests read what rskj refuses and what it keeps, signatures as rskj
takes them, every one-byte v, a wrapped transaction, toGas and
isRemascTransaction. The pool takes a gas price sent with a leading
zero as the same transaction as the minimal one. Tests that built bytes
rskj can't read (an empty gas limit, empty r and s, junk) now build
what rskj's own builder writes, and the REMASC wire fixtures use
Transaction::remasc: their v, r and s of 0x00 are a signature to rskj,
not REMASC's none. Dropping the low-s rule, REMASC's position, the rule
that only signed senders are cached, or the re-encoding, or keying the
pool by the bytes as sent, each fails a test.
A node that runs blocks has no executed head recorded on a new
database. Its state is block 0's, which load_or_build_state builds,
and try_start_sync takes block 0 for the head; but drain_follow_buffer
returned when no head was recorded, so a node started within reach of
its peer's tip, as on a young chain, stored every block that came and
ran none of them until its watchdog moved it to a batch, minutes
later. rskj connects block 1 as it arrives. The drain now counts from
block 0 too; a node that runs no blocks keeps them buffered, as before.

Test: on a new database block 1 is taken up, and a gap above block 0
is re-fetched.
stop() cleared the flag the thread reads after each sleep slice and each
block, and start() set it again: a start in that window spawned a new
thread, and the stopped one read the flag set again and carried on, so
two threads mined. Each start now gives its thread a flag of its own,
and stop clears the running thread's.

Tests, ports of rskj's miner tests that had no counterpart here: the
timed miner's start and stop cycles and its riding out work it can't
build (TimedMinerClient*Test), no fees on an empty block and a notify on
a new parent (MinerServerTest), a refused solution leaving its work on
offer (MainNetMinerTest), a target less than a step below
(GasLimitCalculatorTest), and the commit to parents when the height
mined is a multiple of 64 (ForkDataDetectionCalculatorTest). That rskj
test names the block next to the one its calculator reads, and passes
because its chain gives the two one Bitcoin header; rustock reads the
block rskj's code reads.
rskj's MinerClock.increaseTime returns the offset unchanged for zero or
fewer seconds; rustock added them, so a negative value moved the clock
back and could stamp a block before one already mined. It now answers
the offset as it stands.

Test, in a test binary of its own since the offset is one value for the
process: rskj's MinerClockTest.adjustTimeIfNotRegtest and
clearTimeIncrease, with the moves it ignores between them.
docs/rskj-tests/ holds a table per area of the tests of rskj's release
VETIVER-9.0.4 that name a consensus rule or a network upgrade, 1,239 in
all, each with its verdict: ported, held by a test rustock had, not
applicable, or fixed where its port found rustock doing otherwise.

docs/rskj-test-parity.md links them, flips the rows a test has since
covered, adds the release's co.rsk.mine.gas tests (rustock has no stable
minimum gas price), and counts its rows again.

docs/audit/README.md gives the 70 RSKIPs rsk.zone lists, upgrade by
upgrade, with what was found and, for each fix, whether rskj's regtest
confirmed it, and closes the audit's 348 checklist rows on that reading.
rskj connects a solved block through BlockChainImpl.tryToConnect, which
answers INVALID_BLOCK for any exception while it connects one, a failing
store included, and NO_PARENT for a block whose parent its store lacks;
the pool is answered OK with that. The proof of work is checked first, as
processSolution checks it, then the rest in internalTryToConnect's order:
a block already stored, the parent, the rules, the run, the store. The
node's own miner still reads the error.
merged-mining.md follows complete_and_import as it runs now: the proof of
work first, then the connect in internalTryToConnect's order, and the pool
answered OK with INVALID_BLOCK for a block a rule refuses or the store
fails to take, and NO_PARENT for one whose parent the store lacks.
rskj's miner steers each block's gas limit toward targetgaslimit,
6,800,000 unless set and 7,500,000 on its regtest (regtest.conf), with
forcegaslimit on unless set: a 1024th of the parent's a block, whatever
the parent used. With it off, GasLimitCalculator follows the gas the
parent used: down a step, up by 3/2 of the gas used over 1024, never
under the minimum or past the target, and the parent's kept where that
would move more than a step. rustock's miner kept the parent's gas limit.

--mining-gas-limit-target and --mining-force-gas-limit, and
gas_limit_target and force_gas_limit under [mining], give the two
settings, with rskj's defaults. rskj's GasLimitCalculatorTest is ported
test by test; its validByConsensus reads ethereumj's ParentGasLimitRule,
and the port checks the result against the rule a block is held to too.
The example config files lose two lines node-light.example.toml had twice.
rskj's miner can hold the minimum gas price it steers toward stable in a
currency: a configured price over the price of a bitcoin, fetched from a
web API's JSON (HTTP_GET) or from a contract through a call on the node
(ETH_CALL), refreshed every refresh_rate on a thread of its own, with the
fixed miner.minGasPrice standing in until a price is had. rustock had only
the fixed price, and with --mining-min-gas-price unset each block kept its
parent's; rskj's default steers toward 0.

- The providers (the fixed one, StableMinGasPriceProvider and its two
  kinds) are ported, and the settings are read as rskj reads
  miner.stableGasPrice, from the config file's [mining.stable_gas_price],
  which has no flag, as in rskj.
- A web answer is read as rskj's Jackson 2.15 reads it (its depth and
  number limits, the low 64 bits of a big integer, a JSON pointer's
  escapes), and fetched as Java's HttpURLConnection fetches it for rskj's
  SimpleHttpClient: any 2xx, redirects 300 to 303 and 307 within their
  scheme, resolved as java.net.URL resolves them. Two limits rskj doesn't
  have: an answer past 1 MiB is no price, and a 305 isn't followed.
- A contract's price is the node's own eth_call; with the RPC off, the
  node opens its state to calls for the price alone.
- --mining-min-gas-price defaults to 0, as rskj's miner.minGasPrice.
- Every rskj test in co.rsk.mine.gas and co.rsk.config.mining is ported.
- The lines the last two miner commits added are formatted as rustfmt
  formats them.
The check that refuses to start against a trie that isn't there
(0c83af7) runs after block 0's header is written and before block 0's
state is built, so on a new database it found a chain and an empty trie
and refused: no node started on a new database with the default
--trie-backend single, on any network, other than through --import.

The state to lose is what blocks' runs left, so an empty trie is now
refused only where a block has run (an executed head is recorded). A
node that has run nothing builds block 0's state whatever its trie holds.
The message and the check for a detached trie are as they were.

Test: a new database of mainnet, testnet and regtest passes the check;
with an executed head and an empty trie it is refused, and with a node
in its trie it isn't. With the old condition the first assertion fails.
A new regtest node now starts, and starts again, on the default backend.
rskj's BridgeSupportFlyoverTest has two tests of a Flyover payment over
the locking cap that pays both the active and the retiring federation:
the refund spends each output with its own federation's Flyover redeem
script, in the active federation's format. With both federations
P2SH-ERP the script is last in each input's script; with a P2SH-P2WSH
active federation it is last in each input's witness, the retiring
P2SH-ERP federation's input included, behind a script that pushes the
P2WSH program alone. They were the two of rskj's tests of these rules
left unported; rustock already did as they assert.

Each fails with the refund built without segwit, and with every input
built with the first federation's script.
Upstream's four pull requests since 169ef57: the external IP looked up
over HTTPS when --external-ip is not set (Rustock-dev#365), the node run from a
prebuilt binary in Docker and configured from .env (Rustock-dev#366), libz-sys
moved past its heap-overflow advisory (Rustock-dev#367), and CLAUDE.md on
screening an outside pull request without running it (Rustock-dev#373).

Three files conflicted:

- crates/cli/Cargo.toml: both sides added tokio-rustls (ring, tls12)
  and webpki-roots, upstream's for the external-IP lookup and this
  branch's for --import. They are listed once, under one comment for
  both, beside this branch's serde_json and sha2. Cargo.lock gains no
  package over upstream's.
- deny.toml: both sides rewrote the note on webpki-roots' licence. It
  now names every use: the external-IP lookup, Rootstock's server for
  --import, a stable minimum gas price's web source, and lettre under
  the smtp feature.
- crates/cli/src/main.rs: both sides' modules are kept.
Upstream's tests of the lookup (Rustock-dev#365) start their servers on 127.0.0.1
and look up https://localhost:<port>/ with half a second to spare.
Windows resolves localhost to ::1 first and takes two seconds to refuse
a connection, so the lookup was still being refused on ::1 when the half
second ran out: twelve of the tests failed on Windows, where this
branch's CI runs them.

The servers now listen where localhost resolves first, which the lookup
tries first, on every OS. The test of a lookup with nothing listening is
refused on both addresses, four seconds on Windows, and is given ten.
SDL-A1 added a commit to HORNET-Storage/Rustock that referenced this pull request Nov 18, 2026
cargo-deny's advisories check failed on three RUSTSEC advisories
published against lettre 0.11.23, which the smtp feature builds into
rustock-alerts and so into the node binary:

- RUSTSEC-2026-0560: a man-in-the-middle appends replies to the
  cleartext reply to STARTTLS, and the client takes them for the
  server's replies to the commands it then sends over TLS. A message
  the server rejected can be made to look accepted. Only STARTTLS
  connections are affected (Tls::Required and Tls::Opportunistic, as
  `starttls_relay` sets up, which `TlsMode::Starttls` selects in
  crates/alerts/src/sink.rs); implicit TLS is not.
- RUSTSEC-2026-0561: missing dot-stuffing after a bare LF or CR allows
  SMTP command injection.
- A panic on an EHLO reply line of only white space, a denial of
  service.

All three are fixed in 0.11.24. `cargo update -p lettre` takes
email-encoding 0.4.3 with it, which carries the dot-stuffing fix.

Lockfile only: no manifest changes, and the package count is unchanged
at 469, with nothing added or removed beyond those two versions.
Cargo's re-resolution also moves errno, rustix and tempfile from
windows-sys 0.52.0 to 0.61.2; both were already locked, and both
remain. `cargo metadata --locked` passes and every dependency outside
the workspace still carries a checksum (459 of 469, the other 10 being
the workspace members), which is what the Security workflow asserts.

Found by the advisories job on Rustock-dev#374, not by the daily schedule in
security.yml: that cron has never fired since it was added on
11 September 2026 (0 scheduled runs repository-wide). Filed separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The consensus audit skipped 111 RSKIPs; 11 of them are Adopted or Accepted

2 participants